Pandas Profiling Report

Dataset statistics

Number of variables	11
Number of observations	53940
Missing cells	0
Missing cells (%)	0.0%
Duplicate rows	0
Duplicate rows (%)	0.0%
Total size in memory	4.5 MiB
Average record size in memory	88.0 B

Variable types

NUM	8
CAT	3

Warnings

`price` is highly correlated with `carat`	High correlation
`carat` is highly correlated with `price` and 3 other fields	High correlation
`x` is highly correlated with `carat` and 2 other fields	High correlation
`y` is highly correlated with `carat` and 2 other fields	High correlation
`z` is highly correlated with `carat` and 2 other fields	High correlation
`df_index` has unique values	Unique

Reproduction

Analysis started	2020-11-10 06:10:38.259793
Analysis finished	2020-11-10 06:11:01.705371
Duration	23.45 seconds
Software version	pandas-profiling v2.9.0
Download configuration	config.yaml

df_index
Real number (ℝ_≥0)

UNIQUE

Distinct	53940
Distinct (%)	100.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	26970.5
Minimum	1
Maximum	53940
Zeros	0
Zeros (%)	0.0%
Memory size	421.4 KiB

Quantile statistics

Minimum	1
5-th percentile	2697.95
Q1	13485.75
median	26970.5
Q3	40455.25
95-th percentile	51243.05
Maximum	53940
Range	53939
Interquartile range (IQR)	26969.5

Descriptive statistics

Standard deviation	15571.2811
Coefficient of variation (CV)	0.5773449175
Kurtosis	-1.2
Mean	26970.5
Median Absolute Deviation (MAD)	13485
Skewness	0
Sum	1454788770
Variance	242464795
Monotocity	Strictly increasing

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
2047	1	< 0.1%
6742	1	< 0.1%
12947	1	< 0.1%
14994	1	< 0.1%
8849	1	< 0.1%
10896	1	< 0.1%
53903	1	< 0.1%
49805	1	< 0.1%
51852	1	< 0.1%
37511	1	< 0.1%
Other values (53930)	53930	> 99.9%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
1	1	< 0.1%
2	1	< 0.1%
3	1	< 0.1%
4	1	< 0.1%
5	1	< 0.1%

Value	Count	Frequency (%)
53940	1	< 0.1%
53939	1	< 0.1%
53938	1	< 0.1%
53937	1	< 0.1%
53936	1	< 0.1%

carat
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct	273
Distinct (%)	0.5%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.7979397479
Minimum	0.2
Maximum	5.01
Zeros	0
Zeros (%)	0.0%
Memory size	421.4 KiB

Quantile statistics

Minimum	0.2
5-th percentile	0.3
Q1	0.4
median	0.7
Q3	1.04
95-th percentile	1.7
Maximum	5.01
Range	4.81
Interquartile range (IQR)	0.64

Descriptive statistics

Standard deviation	0.4740112444
Coefficient of variation (CV)	0.5940439058
Kurtosis	1.256635333
Mean	0.7979397479
Median Absolute Deviation (MAD)	0.32
Skewness	1.116645921
Sum	43040.87
Variance	0.2246866598
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0.3	2604	4.8%
0.31	2249	4.2%
1.01	2242	4.2%
0.7	1981	3.7%
0.32	1840	3.4%
1	1558	2.9%
0.9	1485	2.8%
0.41	1382	2.6%
0.4	1299	2.4%
0.71	1294	2.4%
Other values (263)	36006	66.8%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0.2	12	< 0.1%
0.21	9	< 0.1%
0.22	5	< 0.1%
0.23	293	0.5%
0.24	254	0.5%

Value	Count	Frequency (%)
5.01	1	< 0.1%
4.5	1	< 0.1%
4.13	1	< 0.1%
4.01	2	< 0.1%
4	1	< 0.1%

cut
Categorical

Distinct	5
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	421.4 KiB

Ideal	21551
Premium	13791
Very Good	12082
Good	4906
Fair	1610

Frequencies
Length

Value	Count	Frequency (%)
Ideal	21551	40.0%
Premium	13791	25.6%
Very Good	12082	22.4%
Good	4906	9.1%
Fair	1610	3.0%

Frequencies of value counts

Unique

Unique	0 ?
Unique (%)	0.0%

Histogram of lengths of the category

Length

Max length	9
Median length	5
Mean length	6.286503522
Min length	4

color
Categorical

Distinct	7
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	421.4 KiB

G	11292
E	9797
F	9542
H	8304
D	6775
Other values (2)	8230

Frequencies
Length

Value	Count	Frequency (%)
G	11292	20.9%
E	9797	18.2%
F	9542	17.7%
H	8304	15.4%
D	6775	12.6%
I	5422	10.1%
J	2808	5.2%

Frequencies of value counts

Unique

Unique	0 ?
Unique (%)	0.0%

Histogram of lengths of the category

Length

Max length	1
Median length	1
Mean length	1
Min length	1

clarity
Categorical

Distinct	8
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	421.4 KiB

SI1	13065
VS2	12258
SI2	9194
VS1	8171
VVS2	5066
Other values (3)	6186

Frequencies
Length

Value	Count	Frequency (%)
SI1	13065	24.2%
VS2	12258	22.7%
SI2	9194	17.0%
VS1	8171	15.1%
VVS2	5066	9.4%
VVS1	3655	6.8%
IF	1790	3.3%
I1	741	1.4%

Frequencies of value counts

Unique

Unique	0 ?
Unique (%)	0.0%

Histogram of lengths of the category

Length

Max length	4
Median length	3
Mean length	3.114757138
Min length	2

depth
Real number (ℝ_≥0)

Distinct	184
Distinct (%)	0.3%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	61.74940489
Minimum	43
Maximum	79
Zeros	0
Zeros (%)	0.0%
Memory size	421.4 KiB

Quantile statistics

Minimum	43
5-th percentile	59.3
Q1	61
median	61.8
Q3	62.5
95-th percentile	63.8
Maximum	79
Range	36
Interquartile range (IQR)	1.5

Descriptive statistics

Standard deviation	1.432621319
Coefficient of variation (CV)	0.02320056884
Kurtosis	5.739414582
Mean	61.74940489
Median Absolute Deviation (MAD)	0.7
Skewness	-0.0822940263
Sum	3330762.9
Variance	2.052403843
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
62	2239	4.2%
61.9	2163	4.0%
61.8	2077	3.9%
62.2	2039	3.8%
62.1	2020	3.7%
61.6	1956	3.6%
62.3	1940	3.6%
61.7	1904	3.5%
62.4	1792	3.3%
61.5	1719	3.2%
Other values (174)	34091	63.2%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
43	2	< 0.1%
44	1	< 0.1%
50.8	1	< 0.1%
51	1	< 0.1%
52.2	1	< 0.1%

Value	Count	Frequency (%)
79	2	< 0.1%
78.2	1	< 0.1%
73.6	1	< 0.1%
72.9	1	< 0.1%
72.2	1	< 0.1%

table
Real number (ℝ_≥0)

Distinct	127
Distinct (%)	0.2%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	57.45718391
Minimum	43
Maximum	95
Zeros	0
Zeros (%)	0.0%
Memory size	421.4 KiB

Quantile statistics

Minimum	43
5-th percentile	54
Q1	56
median	57
Q3	59
95-th percentile	61
Maximum	95
Range	52
Interquartile range (IQR)	3

Descriptive statistics

Standard deviation	2.234490563
Coefficient of variation (CV)	0.03888966376
Kurtosis	2.80185686
Mean	57.45718391
Median Absolute Deviation (MAD)	1
Skewness	0.7968958487
Sum	3099240.5
Variance	4.992948075
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
56	9881	18.3%
57	9724	18.0%
58	8369	15.5%
59	6572	12.2%
55	6268	11.6%
60	4241	7.9%
54	2594	4.8%
61	2282	4.2%
62	1273	2.4%
63	588	1.1%
Other values (117)	2148	4.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
43	1	< 0.1%
44	1	< 0.1%
49	2	< 0.1%
50	2	< 0.1%
50.1	1	< 0.1%

Value	Count	Frequency (%)
95	1	< 0.1%
79	1	< 0.1%
76	1	< 0.1%
73	4	< 0.1%
71	1	< 0.1%

price
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct	11602
Distinct (%)	21.5%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	3932.799722
Minimum	326
Maximum	18823
Zeros	0
Zeros (%)	0.0%
Memory size	421.4 KiB

Quantile statistics

Minimum	326
5-th percentile	544
Q1	950
median	2401
Q3	5324.25
95-th percentile	13107.1
Maximum	18823
Range	18497
Interquartile range (IQR)	4374.25

Descriptive statistics

Standard deviation	3989.439738
Coefficient of variation (CV)	1.014401958
Kurtosis	2.177695759
Mean	3932.799722
Median Absolute Deviation (MAD)	1670
Skewness	1.618395283
Sum	212135217
Variance	15915629.42
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
605	132	0.2%
802	127	0.2%
625	126	0.2%
828	125	0.2%
776	124	0.2%
789	121	0.2%
698	121	0.2%
544	120	0.2%
666	114	0.2%
552	113	0.2%
Other values (11592)	52717	97.7%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
326	2	< 0.1%
327	1	< 0.1%
334	1	< 0.1%
335	1	< 0.1%
336	2	< 0.1%

Value	Count	Frequency (%)
18823	1	< 0.1%
18818	1	< 0.1%
18806	1	< 0.1%
18804	1	< 0.1%
18803	1	< 0.1%

x
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct	554
Distinct (%)	1.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	5.731157212
Minimum	0
Maximum	10.74
Zeros	8
Zeros (%)	< 0.1%
Memory size	421.4 KiB

Quantile statistics

Minimum	0
5-th percentile	4.29
Q1	4.71
median	5.7
Q3	6.54
95-th percentile	7.66
Maximum	10.74
Range	10.74
Interquartile range (IQR)	1.83

Descriptive statistics

Standard deviation	1.121760747
Coefficient of variation (CV)	0.1957302348
Kurtosis	-0.6181606709
Mean	5.731157212
Median Absolute Deviation (MAD)	0.93
Skewness	0.3786763426
Sum	309138.62
Variance	1.258347173
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
4.37	448	0.8%
4.34	437	0.8%
4.33	429	0.8%
4.38	428	0.8%
4.32	425	0.8%
4.35	407	0.8%
4.39	388	0.7%
4.31	387	0.7%
4.36	386	0.7%
4.4	373	0.7%
Other values (544)	49832	92.4%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	8	< 0.1%
3.73	2	< 0.1%
3.74	1	< 0.1%
3.76	1	< 0.1%
3.77	1	< 0.1%

Value	Count	Frequency (%)
10.74	1	< 0.1%
10.23	1	< 0.1%
10.14	1	< 0.1%
10.02	1	< 0.1%
10.01	1	< 0.1%

y
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct	552
Distinct (%)	1.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	5.734525955
Minimum	0
Maximum	58.9
Zeros	7
Zeros (%)	< 0.1%
Memory size	421.4 KiB

Quantile statistics

Minimum	0
5-th percentile	4.3
Q1	4.72
median	5.71
Q3	6.54
95-th percentile	7.65
Maximum	58.9
Range	58.9
Interquartile range (IQR)	1.82

Descriptive statistics

Standard deviation	1.142134674
Coefficient of variation (CV)	0.1991681062
Kurtosis	91.21455716
Mean	5.734525955
Median Absolute Deviation (MAD)	0.92
Skewness	2.434166716
Sum	309320.33
Variance	1.304471614
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
4.34	437	0.8%
4.37	435	0.8%
4.35	425	0.8%
4.33	421	0.8%
4.32	414	0.8%
4.39	407	0.8%
4.38	406	0.8%
4.4	387	0.7%
4.31	386	0.7%
4.41	384	0.7%
Other values (542)	49838	92.4%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	7	< 0.1%
3.68	1	< 0.1%
3.71	2	< 0.1%
3.72	1	< 0.1%
3.73	1	< 0.1%

Value	Count	Frequency (%)
58.9	1	< 0.1%
31.8	1	< 0.1%
10.54	1	< 0.1%
10.16	1	< 0.1%
10.1	1	< 0.1%

z
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct	375
Distinct (%)	0.7%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	3.538733778
Minimum	0
Maximum	31.8
Zeros	20
Zeros (%)	< 0.1%
Memory size	421.4 KiB

Quantile statistics

Minimum	0
5-th percentile	2.65
Q1	2.91
median	3.53
Q3	4.04
95-th percentile	4.73
Maximum	31.8
Range	31.8
Interquartile range (IQR)	1.13

Descriptive statistics

Standard deviation	0.7056988469
Coefficient of variation (CV)	0.1994212877
Kurtosis	47.08661933
Mean	3.538733778
Median Absolute Deviation (MAD)	0.57
Skewness	1.522422559
Sum	190879.3
Variance	0.4980108626
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
2.7	767	1.4%
2.69	748	1.4%
2.71	738	1.4%
2.68	730	1.4%
2.72	697	1.3%
2.67	649	1.2%
2.73	612	1.1%
2.66	555	1.0%
2.74	548	1.0%
4.02	538	1.0%
Other values (365)	47358	87.8%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	20	< 0.1%
1.07	1	< 0.1%
1.41	1	< 0.1%
1.53	1	< 0.1%
2.06	1	< 0.1%

Value	Count	Frequency (%)
31.8	1	< 0.1%
8.06	1	< 0.1%
6.98	1	< 0.1%
6.72	1	< 0.1%
6.43	1	< 0.1%

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

Phik (φk)

Phik (φk) is a new and practical correlation coefficient that works consistently between categorical, ordinal and interval variables, captures non-linear dependency and reverts to the Pearson correlation coefficient in case of a bivariate normal input distribution. There is extensive documentation available here.

Cramér's V (φc)

Cramér's V is an association measure for nominal random variables. The coefficient ranges from 0 to 1, with 0 indicating independence and 1 indicating perfect association. The empirical estimators used for Cramér's V have been proved to be biased, even for large samples. We use a bias-corrected measure that has been proposed by Bergsma in 2013 that can be found here.

Count
Matrix

First rows

	df_index	carat	cut	color	clarity	depth	table	price	x	y	z
0	1	0.23	Ideal	E	SI2	61.5	55.0	326	3.95	3.98	2.43
1	2	0.21	Premium	E	SI1	59.8	61.0	326	3.89	3.84	2.31
2	3	0.23	Good	E	VS1	56.9	65.0	327	4.05	4.07	2.31
3	4	0.29	Premium	I	VS2	62.4	58.0	334	4.20	4.23	2.63
4	5	0.31	Good	J	SI2	63.3	58.0	335	4.34	4.35	2.75
5	6	0.24	Very Good	J	VVS2	62.8	57.0	336	3.94	3.96	2.48
6	7	0.24	Very Good	I	VVS1	62.3	57.0	336	3.95	3.98	2.47
7	8	0.26	Very Good	H	SI1	61.9	55.0	337	4.07	4.11	2.53
8	9	0.22	Fair	E	VS2	65.1	61.0	337	3.87	3.78	2.49
9	10	0.23	Very Good	H	VS1	59.4	61.0	338	4.00	4.05	2.39

Last rows

	df_index	carat	cut	color	clarity	depth	table	price	x	y	z
53930	53931	0.71	Premium	E	SI1	60.5	55.0	2756	5.79	5.74	3.49
53931	53932	0.71	Premium	F	SI1	59.8	62.0	2756	5.74	5.73	3.43
53932	53933	0.70	Very Good	E	VS2	60.5	59.0	2757	5.71	5.76	3.47
53933	53934	0.70	Very Good	E	VS2	61.2	59.0	2757	5.69	5.72	3.49
53934	53935	0.72	Premium	D	SI1	62.7	59.0	2757	5.69	5.73	3.58
53935	53936	0.72	Ideal	D	SI1	60.8	57.0	2757	5.75	5.76	3.50
53936	53937	0.72	Good	D	SI1	63.1	55.0	2757	5.69	5.75	3.61
53937	53938	0.70	Very Good	D	SI1	62.8	60.0	2757	5.66	5.68	3.56
53938	53939	0.86	Premium	H	SI2	61.0	58.0	2757	6.15	6.12	3.74
53939	53940	0.75	Ideal	D	SI2	62.2	55.0	2757	5.83	5.87	3.64

Overview

Variables

Interactions

Correlations

Pearson's r

Spearman's ρ

Kendall's τ

Phik (φk)

Cramér's V (φc)

Missing values

Sample

First rows

Last rows